AI systems are advancing faster than expected and the next few years could redefine what we mean by intelligence. Will progress continue through scale alone, or are new breakthroughs in algorithms, data and agentic interaction needed to reshape the path ahead?
As models begin to learn, plan and act in digital and physical environments, what kind of intelligence are we creating and how will it shape humanity?
This session was developed in collaboration with The Atlantic.
This is a livestreamed session. Please arrive 15 minutes early as the doors will close at the scheduled time.
At Davos, leading AI thinkers argued that “the next few years could redefine what we mean by intelligence,” but progress will require more than scale. Yoshua Bengio described “scientist AI,” an approach that changes training objectives to make systems “honest in a probabilistic sense,” enabling technical guardrails that predict harms and veto risky actions—while acknowledging society must still set risk thresholds. Yejin Choi emphasized continual, test-time learning to reduce today’s “jagged intelligence,” but warned that continuously evolving systems can invalidate prior safety tests; she called for models that proactively learn and internalize human norms, refusing to learn illegal or harmful behaviors. Eric Xing, who built an academic foundation model (K2), argued current models offer “textual intelligence,” not the “physical,” “social,” and “philosophical intelligence” needed for action, coordination, and self-directed inquiry; he called for new architectures with richer representations and long-horizon consistency. Yuval Noah Harari cautioned that AI will not become human-like—“airplanes” will never be “birds”—and that even “extremely primitive AI” can destabilize society when inserted into human systems like media and finance. The open-source debate sharpened: openness accelerates science and diffusion, but Bengio warned that as capabilities become weaponizable, unrestricted release becomes untenable. The shared conclusion: pair technical “checkpoints” with societal governance, and measure risk continuously, not once.
In AI. The premise is that most of the progress in AI up to now has been through scaling or data, more compute, and that that is still useful. But there are other better things. So I'm going to ask each of our three wonderful panelists to talk a little bit about what they're working on now. By the time we're done with that, our fourth panelist, Yuval Noah Harari, will arrive and he'll join in and try to catch up. So, Yoshua, you are working on scientist AI, which is incredible. Explain what it is and how it's different from previous paradigms of AI.
Thank you. Thank you. So what's motivating the scientist AI and also the new nonprofit I created to engineer it called Law Zero, is, how it it addresses the question of reliability of the AI systems we are building, especially the genetic systems, how it deals with the issue that current AI systems can have goals, subgoals that we did not choose and that can go against our instructions. And this is something that's already been observed. And it's, you know, even more prevalent in the last year across a number of experimental studies, but also in the deployment of AI, for example, with sycophancy. It's an issue that is, kind of very concerning when you look at the behavior of self-preservation, where AI's don't want to be shut down and they want to evade our oversight, be willing to do things like blackmail in order to escape our control. So, even things like preventing, misuse the companies put monitors and guardrails, but somehow this still doesn't work really well enough. And the core of our thesis is that we can change the way that AIS are trained. So it could be the same kind of architecture, but the training objective and the way we manage the data, is going to be such that we obtain, guarantees that the system will be honest in a probabilistic sense.
Okay. So how do you do that?
How do you do that? So the core of the idea, which is connected.
To do it with my kids.
Yes. So the core of the idea which is behind the name, is take as an inspiration not to imitate people, but to imitate what science at an ideal level, is trying to do. So think about the laws of physics, the laws of physics. Physics can be turned into predictions and those predictions will be honest. They don't care about whether the prediction is going to help one person or another person. So it turns out that it is possible to define training objectives for neural nets so that they will converge to what something like scientific laws would predict. And then we get something that we can rely, for example, we can rely on to create technical guardrails around agents that we don't trust. So if an agent is proposing an action, for each action that the agent proposes, harness predictor could tell us whether that action has some probability of creating a particular kind of harm. And, of course, veto that action if that's the case.
But you still are then going to be required to put in some threshold of when it will take that action. Right. If it has a percentage odds of harm of more than 1 in 10 or 1 in 1000, wherever you put it, you still have some human concern. You still have some potential harm.
Absolutely. So when we build a nuclear plant, we have to decide where we put the threshold.
Oh, so we're okay, right?
And, for nuclear plants, it might be 1 in 1,000,000 years that something bad is going to happen because it's so severe, depending on the kind of harm that we're trying to prevent society, not AI's have to decide where we put those thresholds. Right.
I've always thought it was interesting that, for most things, we'll accept like a one in a 10 million chance of nuclear plant exploding. But we continue to build AI, even though general predictions that might wipe out humanity are like 10%. All right, Eugene, why don't you talk a little bit about some of your work in continual learning? And you, of course, have been a brilliant critic of scaling laws for a long time, including on a panel last year with Yoshua. So tell us what you're working on now.
All right. So let me step back a little bit before I do continue learning. You know, right now AI is like a super impressive, but it's a little bit jagged intelligence, right, in that it's amazing at bar exams and, you know, some of these, like, really difficult, International Olympiad problems yet, you know, you're not going to rely on it for, you know, doing your tax return or even like, making some important transactions because it may not be able to click the right button on your computer. So why is it that it's because right now, the way that we train AI Llms generative AI is two data dependent and it's one time training, and then you deploy it and it may or may not make a mistake. So here are a few really important things that I think we need to solve in order to get to the next, next intelligence. So the first first of all, continual learning. So, you know, like one on one machine learning 101 is that you separate out training from testing. It's almost a sin to mix the two. But human intelligence is not like that. From the day one, a baby is born. It's in the deployment mode. It has to, you know, figure things out. It's real life so humans can learn during the deployment time. And we need to somehow figure out how to ensure AI can learn continuously during the test time. So it's test time training that I'm working on. Another angle that's really important is that currently the one of the key reasons in my mind why AI is unreliable sometimes. And, you know, we need to worry about the safety concerns as well, that, you know, for example, pay per click, you know, scenario where you, you ask llms to generate as many paperclips as possible, it might kill all of us in order to produce one more paperclip. Right? So in order to avoid that kind of situation that's harmful for humans, AI should really figure out how the world works for the sake of learning how the world works, as opposed to just passively learning, you know, whatever, data that's given to us. So I think fundamental challenge here is that llms learn passively as opposed to proactively. It's not really thinking for itself. It's just trying to memorize all the texts given to us and then try to solve all the math problems given to us, as opposed to us humans being curious about how the world works and trying to think for ourselves. And then lastly, it's a way to data dependent whatever data, you know, whatever data is reached, it works. Whatever data is not reached, it doesn't work. That's how things are right now. And then, you know, safety is hard because we have to create all the safety data and, you know, red teaming jailbreaks these are not area of domain where there's a lot of data. So, in order to fix this problem, I think we need an entirely different learning paradigm where it's really about thinking for itself, almost trading off data with compute. So, you know, learning with way less data but with more mental efforts.
Quickly aging. But if you have continual learning, doesn't that open up a whole new spectrum of problems? Like right now, you build a model, you run a bunch of tests, eventually you refine it. A few months later, you do it again. If it's continuously learning, how does that not become just suddenly infinite? Right? I mean, if you are learning from every answer and you're giving feedback and like a baby's like in its crib, it's walking around, it's contained. But if you have, you know, a few billion people using a model at any time and it's learning constantly, doesn't that open up whole whole new vectors of wonder, but whole new vectors of problems?
Yes and no. So it could, in theory, in a long, long term. But, it's just so far off in my mind in the sense that humans can also continually learn. But there's a limit as to how much we can really reach.
But another problem is that, after the system has evolved sufficiently through this continual learning, all the safety tests that we did previously may not be valid anymore. So I think there's a real safety risk that you're pointing to.
Yes. So my hope is that, if AI is trained correctly from day one, so that it really understands human norms and values, not just math problem solving, but human norms and values such that it will build its worldview and everything else on top of it. And then it's going to behave based on that.
And how do you deal with reward hacking? In other words, even if it understands human values, it might have optimized something that is not quite what we want.
I agree.
I agree.
With that. Yeah. So reward hacking implies that, you know, we just go with reinforcement learning and that's all we got. No it shouldn't be all we got. No human being is, optimizing for one reward for the rest of their life. Right. Like, we have so many different goals that are at us and we make some sacrifice. You know, I might want to do something, but I might not do it because I respect other people. Right. So AI should be exactly this, that they should understand that values are at odds in real life, in human life, and that it needs to know how to make the trade offs such that it's going to not violate laws, it's not going to harm people. And whenever it's not clear what to do, because there are always situations in which it's not clear what the gold answer is, it should consult with humans and release the decision making to humans.
All right. We're going to move to Eric. Eric, you have recently built a big new, big new model, K2. I think you've had a whole series of innovations in it. Explain what you've done, what you've done that's novel. And what's different from the amazing stuff that Eugene and Yoshua are working on?
Well, I have thoughts on my boots. So, yeah, as Nick just mentioned, we at MSI are among the few, maybe the only university that is actually building those foundation models from scratch, from scratch. Meaning that you gather your own data, you implement your own algorithm, you build your own machine, and then you train from top to bottom, and then you release and serve the whole process. I thought that it's important for academics to be a player like this, so that we can share the knowledge to the public, so that people can study many of the nuances in building this and also understand the safety and risk issue. In fact, I want to say that it is by no means easy. It's very, very difficult. In fact, I almost want to say that AI systems and softwares are actually very vulnerable. They are not very robust and they are not very powerful. You remove one machine from the cluster, you can crash the whole thing already. Now, what I'm building right now is of course to improve AI performance, but I want to maybe add on your question, a comment, you know, on, what do we mean by intelligence and how to break it down? Because if I tell my engineer, say, hey, build a software that is intelligent, they don't know what to do. So many people have different opinions on intelligence. There are Nobel Prize winners, you know, in economy who may not do very well in their stock inspection and their wife may do better than them. You know, that's actually reflecting different levels of intelligence and different utilities. In my opinion. What LM right now is delivering is a limited form of intelligence. I would call them maybe texture intelligence or maybe visual intelligence, which is actually on a piece of paper in the form of language or maybe video, but these are like book knowledge if you want to put it on action. I was actually doing hacking a week ago in in the Austrian Alps. I do the Gpts, I do the Google, I got all the guides and even Google maps in my hand. When I walk to the mountain, you still cannot rely on paper. You have to rely on yourself. You know, you have all these unexpected situations. Now is too deep and the weather is no good and you cannot see the path anymore. What do you do? So this requires already a new type of intelligence that is not available right now, in which we call physical intelligence. And that's actually where people hear about the topic of word models. Word model is about understanding the word able to generate the plans and the strategies and the sequence of actions purposefully so that you can execute it and you can actually deploy it. And also you can adapt to changing environment. But still, this is a not necessarily the smartest thing that we could imagine because, you know, I would call the next level beyond physical intelligence would be social intelligence. You know, right now we don't actually see two llms collaborating yet. They don't really understand each other in the form that we humans do. Right? There is no definition of a self. What is my limitation? What is your limitation? How can we divide the job into 2 or 100 so that we can, you know, break them into parts? Therefore you cannot you can never ask LLM in our model to help you run a company or run a country, because they don't understand this kind of nuances of interactive behaviors. I would put in fact, also a last layer of intelligence that is still even further, which I would for the sake of for the lack of good name. I call them philosophical intelligence, which is that SLM or AI models itself queries to discover the world, to look for data and to learn things, and then to explain without, you know, being asked to explain. That's probably where Josh is very, very concerned about, because that's where you start to see definitively some sign of identity and agency. I want to say that we are not there yet. We are very far from there. Even the current physical model, the world model, is very primitive because it is primarily rely on a wrong architecture that is directly offspring of the LM. So what my work is involving right now is to come up with new architectures, which represents the data, do the reasoning, and also do the learning using different ideas. People may have heard about the architecture of Japan, right? It is an architecture behind many of the current world models. We have a model called JLP which does the following. First of all, your representation of knowledge needs to be richer, need to be containing both continuous and symbolic signals so that you can reason at different level of granularity. And secondly, you need to have the right architecture which can carry a long way. People play with the solver, probably have that experience. How many seconds of video can generate? Maybe 10s maybe a minute. It's not because they run out of memory, it's because going beyond 1 minute or 10 minute, you don't have the ability to track consistency to reason consistently. In fact, you can try a very interesting experiment. You just ask The Sorrow or maybe Gemini to generate you 360 degree of round view around you and then turn back to your degree. Zero did you see the same thing or not? It is not guaranteed. That's actually a lack of consistency already in the system. And then stateful representations, you know, and also there are things like continuous learning paradigm is a problem. Right? Right now all models are in the form of what we call passive learning. You feed the data and the and then the model will be trained on those data. In machine learning, we knew in the past a new paradigm called active learning or proactive learning, where the system should hopefully be able to identify where they want to learn more by using asking for more data. But we are not yet there. Not long go out and looking for data and create data themselves. So I think, you know, AI, as of now, in my opinion, still is a very primitive age. We have a lot to do to really get it to work.
Well, Yuval, this is the handsome man at the end of panel, Yuval Noah Harari. You probably already know that. Welcome. How are you? He's late for the same reason that, like, everything is complicated in Davos, which is geopolitics. Apparently Macron went late. I pushed his panel back. So here we are. Yuval, what we've been talking about is different paths. New research. Yoshua has been doing that. Eugene's been doing. You just walked in. What Eric has been doing lots of different promising ways to make AI go faster. I'm going to just ask you a philosophical question, which is, do you think that as we look for new models of AI, we should be trying to make it more like the human mind, or less like the human mind? This is something you've written about beautifully, but I haven't heard you talk about this in the last while.
No, I think it's.
Completely different from the human mind. The whole question of when will AI reach the same level of human as human intelligence? This is ridiculous. It's like asking, when will airplanes finally be like birds? They will never, ever be like birds.
And they shouldn't.
Be and they shouldn't be. And they can do many, many things that birds can't. And this will be the same with AI's and humans. They are not on the same trajectory behind us. They are on a completely different trajectory, for better or for worse. I'm very happy to hear that it is still that AI. I'm again, I'm not sure to what extent we can rely on it, how long it will continue, but the fact that AI, for instance, cannot cooperate. So far this is wonderful news. I hope it's true. I hope it will remain like that. Otherwise we are in very, very deep trouble. For me, the lesson from from history about intelligence. You don't need a lot of intelligence to change the world and potentially to cause havoc. You can change the world with relatively little intelligence. And the other thing we've learned from human history about intelligence, I'm not referring to anybody in particular. And the other thing we've learned about intelligence is that the most intelligent entities on the planet can also be the most deluded. Human beings are by far, so far the most intelligent entities on the planet and the most deluded. We believe ridiculous things that no chimpanzee or dog or pig would ever dream of believing like that. If you, go and kill other people of your species after you die, you go to heaven and they'll live blissfully ever after because of the wonderful thing you did that you killed these other members of your species. No chimpanzee will believe that, but many humans do. At least where I come from. And, you can again, when I say that you can change the world with relatively little intelligence. Humans have already done much of the of the hard work for the AIS. Like if you drop an AI in the middle of the African savanna and tell it take over the world, it can't, how will it do it? Impossible. But if first you have these apes who build all these bureaucratic systems like the financial system, and then you drop the AI into the existing financial system and you tell it, okay, now take this over. That's much, much easier. The financial system, you don't need motor skills. You don't need even to understand the world an AI can understand. The financial system is the ideal playground for AI. It's a purely informational like to train, train AI to make $1 million, create a million AIS, giving them some seed money. Let's see you make $1 million. Now, if you have a few AIS that succeeded in doing that, replicate them. What happens to the world if, more and more of the financial system is shaped by AI that developed, even though they can't walk down the street, they know how to invest money better than humans. It's a very, very limited intelligence. But again, think about social media. Social media is run to some extent by extremely primitive AIS, these algorithms that control our news feed and so forth. Look what they did in ten years. We created a human system media, and then we introduced the AI into our system. And it's an informational system. And they took it over and they, to a large extent wrecked the world. They are not the only reason for for the mess now in the world. But if you think about what extremely primitive AI did within the human created system of of media, then.
Well, I'm going to I'm going to move this to Yoshua because he in fact has invented AI or is working on inventing AIS that if dropped into the financial system and told to wreck it, would not be able to correct. That's the hope. Respond to Yuval.
I want to add something. Going back to your first question, connecting humans and AIS and whether we should build AIS at our image. Yeah. And indeed there are quite different from us. The problem is we interact with them. Many people interact with them with the false belief that they are like us. And, the smarter we make them, the more it's going to be like this and there will be people who want to make them even look like us. So it's going to be video first, eventually, maybe physical form, but it's not clear that it's it's good in many ways in terms of how you know, humanity has developed norms and expectations and psychology that work because we interact with other humans. But AI's are not really humans. For example, they could be immortal, right? Once an AI is created, in principle, you could just copy it on more computers. And we can't do that with our brain. As Geoff Hinton has been highlighting many times, and many other differences, like they can communicate with each other a billion times faster than we can do with between each with, with, with ourselves. And so there, you know, there's going to be this illusion that we build machines that are like us, but they're not. And this is a dangerous illusion that could lead us to take wrong decisions. The problem part of the problem is the scientists themselves, like, in the last 40 years that I've been working on AI, in the whole community, really, we took inspiration from, human intelligence. Right. Like you were talking about continual learning because we're good at that. And we see that it's lacking in in AI, and that's fine. That's how research has been moving. But I think we also have to think of what's going to happen when this, gets to be deployed in society more and more and people will anthropomorphize and do weird things.
It's an amazing question. Let's, let's, let's move this to a, to a topic that I think connects to this pretty well, which is, back to the architectural questions or foundation questions, which is the question of open source. And there's actually been more and more discussion here in Davos, in part because Europe is recognizing the need to counterweight to the US AI models. Eric, you're building open source models. You have strong views on them. Yoshua you have strong views on them. Eugene, why don't we start with you? What do you think of the notion that it would be good if there were many more open source models that we all started to use, as much as we use the large foundation models?
Yeah. So the way that I like to think about open source is democratization of generative AI, which is a powerful, powerful tool. And what I mean by democratization of generative AI is that it should be AI should be of human for human by humans. AI is of human because it's really drawing from the internet data. That's the artifact of human intelligence. It reflects our values. It reflects our knowledge, by the way, values, including horrible value, you know, that we do to each other. It happens to be on the internet. And so AI picks up on that. There are sci fi movies in which AI kills us all, and as a result, that's what I might actually say because it's written on the internet. AI should be for humans in that, you know, humanity at large and all of the humans, not just some humans who happen to be in power. I deeply care about this, that AI should be really for all humans. And by the way, worse than AI for some humans is AI for humans, or humans for AI. Even worse, it's really good to think about how we build and design AI so that we work on problems that could really make humanity better, as opposed to, you know, just increase subscriptions and, win the leaderboard. And then lastly, AI by humans, what I mean by that is that AI should be created. AI should be able to create it by, you know, different countries and different, not just private sectors, but public sectors and non-profit organization academia. The reason why I think about this way is that, well, I'm US citizen now, but I used to be a Korean person, and it's a very wonderful thing if we we know how to create this even from Korea or from other countries, as opposed to them having to just rely on a country or two providing all the services for them.
But would your would your goals be satisfied if Korea had a closed foundation model? Or do you want there to be a universal open model that everybody is able to contribute to? Eric, you agree.
People can choose to close or open, but the reason why, for the time being, I really support open source is because, it takes too much of resources to build something really, really good fast. And so unless you're capable of really, making very large data centers and on lots of GPUs really fast, it really helps to help each other to, share the scientific knowledge and everything so that, the development goes much faster. And by doing so, by the way, we can make small models much more powerful so that, a lot of organizations who cannot afford as much can build llms that serve just their, their needs, not like General llms that can do everything, but something that really serves a business need really well.
Right, Eric? So you nodded at one point and shook your head at another point. So I need you to respond quickly here.
Yeah, I think open source isn't really the goal. It is basically a philosophy or a way of doing things which come very naturally with science, with any of the scientific.
What do you mean? It's not.
The goal?
Like it's not like you're not doing it for the sake of open source. You're doing it because it's a more efficient way to reach the outcome.
No, no, no, it is really a almost like a responsibility or a natural style of doing the research of AI. You know, in fact, also pragmatic values. For example, I often ask, do you prefer there is only one carmaker in the world that makes you feel safer, or you actually see 10 or 100 is better, right? Open source basically is about sharing knowledge to the general public so that people can use it. Also, people can study it and understand it and improve it. Of course, the assumption is that this technology, you know, is not an evil, okay. It's not something that you really want to get rid of. I don't think technology itself, by definition, any technology is evil. It's really about the people who use it in the wrong way. But by closing sourcing it, you don't actually stop that. So open sourcing, you know, over the benefit from open sourcing in my opinion, over weights, closing it because, you know, first of all, you cannot stop, you know, the threat of using it. And secondly, by opening it, you actually, you know, are promoting more adoption and more understanding of that. I also want to go back to the issue that Josh just mentioned about, the the impersonalization of human technology creates the risk. Well, that is how we see it now. It's kind of also, you know, implicitly assuming that the way human beings do not learn from the new experiences in the past, if you look at the history, there are many magical inventions which may make certain population godlike. But then after some time, people actually get comfortable with it and start to form better judgment and also better understanding. I think the way of really making people safe and comfortable and coexisting nicely with AI is to use more AI and also get quickly adapt to it. It's like you are in a natural environment, you have the virus and so forth. Of course you want to, think about stopping it, but sometimes in the nature choose to let you coexist with the virus so that you become stronger. You know, there are some risks, some casualties. But as a population, as a society, we together evolve stronger.
Yeah. I have a question for.
You as a university professor. You know, I've been promoting open source and of course, open science for all my life. But, if you start asking ethical questions, then you, you, you know, at some point you start hitting a problem, which is some knowledge can be dangerous when it is available to everyone. So I'm going to give you a simple example. Biologists are working on how to create new DNA sequences that can actually create new viruses that don't exist. And if you know a sequence that gives rise to a virus that could kill half of the planet, should you publish it, and the answer should be obvious in this case, right? So current AI systems that are open sourced are net positive. It helps us. It helps safety. It helps democratization of AI. And I'm as worried as you are about concentration of power. I'll come back to that. The problem is if the capabilities of AI continue to grow along the directions that we've been talking about, at some point we end up with AI systems that are like, well, not the sequence itself, but the machine that can generate the sequence that can kill half of the population. So when AI reaches that stage, we should not just, you know, give it to everyone because there are a lot of crazy people. There are dangerous people. There are people who want to use it, you know, for, destroying their enemies and military ways. So we should be very careful when we reach a level of capability where AI can be weaponized. Now, I agree about the issue of concentration of power, but there are other ways than open sources. When we get to that point where AI can be weaponized, I think. And before we get there, we need to think about it. We should really think of how we can manage, a few, not just one, a few AI systems that will be dangerous in the wrong hands and where the power to control these things will be decentralized. Right. So what we don't want is one entity, one government, one corporation to dictate, you know, what the world should be. But I think that there are solutions to this. And we have experienced this sort of thing in the, the international arena, with international treaties, what we've done with nuclear weapons, what Europe has done with the EU and so on. So I think that there are solutions and we should think about ways to both avoid catastrophic use and abuse of power in the hands of just.
A few. This is I want to bring in Yuval here because this is like this is an amazing philosophical question, right? There's a incredibly powerful technology. Are we safer if everybody's contributing to it and everybody has a say over it, but everybody kind of has access to it? Or are we safer if a relatively small number of people can be controlled or answerable to governments and are all here in the somewhere in the Congress Center, have control of it? Have we ever faced this historically? Yuval, has there ever been a moment like this and what was what happened?
I wasn't okay, sorry.
I think the main point is that we just don't know. We are at a point when we are conducting this huge historical experiment, and we just don't know. The key question for me, how do we build a self-correcting mechanism into it? How do we make sure that if we get the answer wrong, we'll have a second chance? And the model for me is the last big technological revolution, which is the industrial revolution. When the Industrial Revolution begins in the early 19th century, nobody has an idea how to build a benign, a good industrial society. This immense new power, steam engines, railroads, steamships, how do you use them for good? And different people have different ideas and they experiment. And European imperialism was one experiment. Some people say the only way to build an industrial society is to build an empire. You cannot build an industrial society on the level of one country, because you must control the raw material and the markets. You must have an empire. Then you have people who say it must be a totalitarian society, only a totalitarian system like Bolshevism or like Nazism. The immense powers of industry can only be controlled by a totalitarian society. Now, looking back from the early 21st century, we can say, oh, we know what the answer was. We think we know. It took 200 years of terrible wars and hundreds of millions of casualties and, you know, injuries that are not healed even today. To find out how to build a benign industrial society. And this was just steam engines. Now we are dealing with potentially superintelligent agents. Nobody has any experience with building a hybrid human AI society. We should be a lot more humble in the way that we think we know how to build it. No we don't. How do we make sure? I don't know what the answer the question is, how do we build a self-correcting mechanism? So if we take the wrong bet, this is not the end.
I want.
To bring the conversation from philosophical back to more like a practical part, because it's about where the checkpoint should be, right? You talk about a dangerous virus. A virus actually not easy. You know, the idea of a nuclear bomb, for example, is published somewhere. Can Google it, but you cannot build it because you need to get the materials, you need to get the labs. There are a lot of checking points already. You know, we learned from generations and centuries of governance and regulation and human practices. We set in many places already. After all, AI is a piece of software. It is software living in the computers. And when it does the physical harm, it needs to go out of the computer. That's already what extra.
Checkpoints humans can do it for, for the AI. And eventually there will be robots that will do it.
And humans are subject to checkpoints as well, right? Virus, on the other hand, does not.
But let me let me ask you this. Since since this is all this panel is all about, like how to best construct the next generation of AI, probably all agree here on this panel that we we want lots of checkpoints and good checkpoints. We maybe disagree on whether we have enough right now. What is the sort of methodology or architecture of AI that has the most checkpoints? Eugene, you got one right there.
Yeah. I mean, I have a proposal to handle this situation better. I think fundamentally the the problem is that AI is too dumb. It's going to learn on any data that you give to it. And if you happen to give data about how to do cyber attacks or how to generate bio weapons, it's just go ahead and learn from it, right? That's the fundamental challenge we're dealing with. On the other hand, if we build AI, maybe following Russia's AI scientists direction, that really learns, think for itself and really acquire human norms, understand that that's what it should really abide by. And then when it reads the training data given by some other human, it refuses to learn. When it knows that this is illegal, it refuses to learn. And by the way, that's what humans also do. Like a lot of us, of course, there are people who want to do bad things, but a lot of us, if I give you how to kill humans, I mean, like, you know, through bio weapons, would you, you know, internalize it for yourself? No, because you you don't want to act on it. So I think we may need to rethink about how we design AI training algorithms such that it has more agency about like how to choose what to learn.
Or to just not train on Reddit at all. Yasha, I.
Just want to mention that because we've been talking about the technical aspects of these questions right now, the way we design AI systems, there is no boundary between, data and instruction. So in normal programming, it's two different things, right? So the program will read files and then there's the code itself. And the programmers write the code. And they know that whatever is in the files, the behavior is going to be according to the code. With the way that we're building our eyes, there's no distinction. And so that's the reason why it's so easy to, in the data, put instructions. That's how you get jailbreaks right. And other security issues that we have with AI. And so I think that in order to get more safety from the AIS, we, we need them to understand the distinction between what we want and what is instructed in a way that's been socially kind of regulated. So who decides what the norms are and so on. And what it reads as data, what when it has an interaction with a user, we don't want the user to be able to make the AI do anything that it that they want, for example.
So is that like a set of like master controls you're trying to build? Is that like a no.
So in the scientist AI the the way that we're doing this is we're training the AI to make the difference between what people will say will. Right. Which could be motivated, which the AI should not take as truth or what it should be necessarily doing. And, other forms of information which, contains underlying truths or underlying causes of what is being seen. And that second channel is one that is trustworthy, where we, you know, we don't necessarily give that access to, anybody using the AI, for example. But it's also a way to make sure we get AIS that understand the difference between what people will say and what what is actually the cause of what they say, and if what they say is true. So they get honesty.
So we have just a few minutes left. We've talked about a lot of new architectures. We've talked about some new genetic systems. We've talked a lot about open source. We've talked about continual learning. We've talked about different ways of looking at data. And we've kind of talked about all the new systems as though they're good. Are there any sort of new architectures or methodologies that people are excited about, maybe ones that we've talked about on stage that you think are actively bad and that we should not pursue? Maybe Eric and Eugene.
What do you mean by bad architecture?
The consequences are bad or the performance are bad.
Either. Either works fine.
I think. In fact, maybe that is even compatible with what Josh is worried about building a system that is not in the closed loop fashion that you purely do, thought experiments and the embedding internally in some kind of latent representations and complete all the training before emerging to the real world to validate, in my opinion, is a bad system because first of all, performance wise, there isn't really, enough checkpoints to even control, and visualize or understand any of the, risking points. And also, it is very hard to connect the system to a action conditioning point so that you can steer it, you can navigate, you can manipulate. On the other hand, you know, it is going to consume, data and energy and resource and money, you know, for too long before you actually see the outcome. I'm not going to name any specific instance of this architecture, but it's actually pretty prevalent that people sometimes believe that I don't need to really, you know, compare, you know, the content from AI system with real world data constantly before I achieve a superintelligence. And secondly, I also think the current learning paradigm, which I totally agree with Josh and Jean about, is, a very, very primitive and maybe unproductive one. You know, the data, you know, right now is, really, you know, the master of the algorithm and of the system and the system itself, is basically a one shot learning in a sense. You train it and then when now I'm using GPT or any models, they don't actually learn from that experiences. And just like ourselves when we in the conversation, I'm already learning from both of you and all of you new points. I enjoy that, and the AI system isn't built for that kind of functionality yet. And can you imagine that a system of that kind of dumbness can become superintelligent and come back and go after us? I just don't feel that the dots can connect. You know, it doesn't have that kind of a task oriented type of data that guides you, you know, beyond just pattern matching, but actually do the reasoning and so forth. So if our goal is to build smarter and more powerful systems, there are needs to explore new architectures. Of course, there is a separate issue about how do we measure the risk. I don't really know the exact answer, but I want to actually hear Josh, your opinion is the solution to not doing that or do that with a very, very, conscious and quantitative kind of approach to measure the risk, to experiment with all the scenarios.
And then very quickly.
Yes, we we need to measure the risk on the fly, not just once when we evaluate those models. And, we need to make sure that we also have the right societal infrastructure. So even if we knew how to build really safe systems, there are lots of bad things that can happen because humans are humans. And so, you know, we need technical guardrails and we need societal guardrails.
Absolutely.
Yeah.
All right. Let's wrap this up. Yuval. Your book came out, Nexus came out about a year and a half ago. You had some real concerns about AI. You've just been here with three of the smartest, most influential AI researchers. And, in the world, they've won every prize imaginable. Do you feel like we're getting onto the right track, or do you not?
I think.
We are thinking on different time scales that when people a lot of the conversations here in Davos, when they think when they say long term, they mean like two years. When I say long term, I mean like 200 years, it's like, again, it's an industrial revolution. The first commercial railway has been opened between Manchester and Liverpool in 1830. This.
This is now 1834, 1835. And we are having this discussion, people saying the industrial revolution is moving so slowly. They told us that railways and steam engines will change the world. So what? So a few people are going between Manchester and Liverpool didn't change anything. This is all science fiction because the timescale that we have no idea, even if the all progress in AI stops today, the stone have been thrown into the pool but it just hit the water. We have no idea what are the waves created, even by the AI that are already have been deployed, say a year or two ago? Social consequences are a completely different thing. You cannot run history in a laboratory and see what are the social consequences of. You can test for accidents. You create the first steam engine. You can test for accidents. You cannot test what will be the geopolitical implications of the cultural implications of steam engine in a laboratory. It's the same with AI. So, it's just far too soon to know. And I'm mainly concerned about the lack of concern that we are creating. We are deploying the most, maybe the most powerful technology in human history. And a lot of very smart and powerful people are worried about, you know, what will the investors say in the next quarterly report? They think in terms of a few months or a year or two.
Joshua, just quickly, I want to thank Yuval Harari because he's talking about lack of concern. And I've started a new organization, a nonprofit that's trying to implement the scientists AI. And Yuval has graciously accepted to be on the board. We need people like him to look with an independent oversight on on what we be doing with AI, on with society in the coming years.
All right. Time scale 00000. Thank you so much. This was an amazing panel. You're all absolutely wonderful. Thank you for the work you do. And thank you for participating here.
Thank you.